Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus performs an inspection on an assembly component with use of one or more images of an assembling work, and includes a work determination unit configured to determine, from the one or more images, one or both of a start frame and an end frame of assembling work captured at a change point of the assembling work, a selection unit configured to select frames to be inspected from the one or more images based on a result of the determination by the work determination unit, and an inspection unit configured to perform an inspection on the frames selected by the selection unit.

BACKGROUND Field

The present disclosure relates to a technology to perform inspectionusing a captured image.

Description of the Related Art

Manual assembly of a product in a factory includes a number of processesand work, and a defect, such as an omission and a mistake in mountingcomponents, may occur. Such a defect leads to a defective product.Therefore, it is necessary to perform a mounting work again in a casewhere the defect occurs.

As a technology to check the defect of the product, a method calledimage inspection, in which an inspection using image analysis isperformed on an assembled product, has been practically used. In theimage inspection, a correct answer image in a state where componentshave been correctly mounted is previously prepared, and an image of anassembled product to be inspected is collated with the correct answerimage to achieve the inspection.

As an approach to determine the defect of the assembling work, atechnology to determine the work defect in real time through analysis ofmotion of the worker has been developed. It is, however, difficult toguarantee that the assembly has been correctly completed only by theanalysis of the motion of the worker. Even if the worker has performedpredetermined work, a possibility of a mistake, such as the omission ofa component to be mounted in the work and deviation of a mountingposition, remains.

Japanese Patent Application Laid-Open No. 2013-148380 discusses an imageinspection system that detects a defect of a circuit pattern on asubstrate. A plurality of different inspection tasks is sequentiallyexecuted while the substrate is rotated and moved. The inspection tasksare different in coordinates of an inspection target area, specificinspection contents such as cell comparison, inspection parameters, etc.The plurality of inspection tasks are performed on a plurality ofpreviously-determined inspection areas at previously-determinedinspection timings. The plurality of inspection tasks are executed atdifferent timings depending on a state of an inspection object. Theinspection timing is controlled by an apparatus.

Japanese Patent Application Laid-Open No. 2016-205836 discusses atechnology to perform an inspection of a product with use of a cameramounted on a helmet of a worker. The worker can perform the inspectionof an object while performing the other assembling work unrelated to theinspection. Since there is a plurality of types of the inspectionobjects, imaging and an inspection corresponding to the type can beexecuted by reading identification information (barcode) to identify thetype of the object.

For example, in a product in which a large number of components aresuccessively mounted as with a multifunctional peripheral, a cover isoften mounted in the next work after the components are mounted.Therefore, if the inspection is not performed before the completion ofthe assembly, an inspection whether the components have been correctlymounted cannot be performed. Further, if a defect is detected in theinspection after the all of the components are mounted, a large amountof rework, such as disassembly, is necessary. Accordingly, it isdesirable to perform the inspection as early as possible after each ofthe components is mounted. In a case where the assembling work ismanually performed, however, a time necessary for each work is notconstant. Therefore, the timing suitable for the inspection of thecomponent mounting is not fixed, accordingly it is difficult to performimage inspection during the work before the assembly is completed.

In the technology discussed in Japanese Patent Application Laid-Open No.2013-148380, appropriate inspection timing cannot be previouslydesignated with respect to the manual assembling work.

In the technology discussed in Japanese Patent Application Laid-Open No.2016-205836, the inspection contents for the product are independent ofthe assembling work performed by the worker, and the inspection cannotbe performed at timing suitable for the work carried out by the worker.

SUMMARY

Some embodiments in the present disclosure are directed to a device thatperforms an inspection corresponding to each component of an assembledproduct at appropriate timing with respect to a mounting work of eachcomponent.

Further features of various embodiments will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B each illustrate a system configuration example accordingto an exemplary embodiment.

FIGS. 2A to 2E illustrate an example of a usage scene of the system andan example of a relationship between work execution and an imageinspection, according to an exemplary embodiment.

FIGS. 3A to 3D each illustrate an example of a data format according toan exemplary embodiment.

FIG. 4 is a flowchart illustrating a flow of entire processing accordingto a first exemplary embodiment.

FIG. 5 is a flowchart illustrating a flow for selecting an inspectiontarget candidate frame according to an exemplary embodiment.

FIG. 6 is a flowchart illustrating a flow for selecting the inspectiontarget candidate area according to an exemplary embodiment.

FIG. 7 is a flowchart illustrating a flow for selecting a finalinspection target according to an exemplary embodiment.

FIG. 8 is a diagram illustrating an occurrence of shielding in an areaaccording to an exemplary embodiment.

FIG. 9 is a flowchart illustrating a flow of entire processing accordingto a second exemplary embodiment.

FIG. 10 is a flowchart illustrating a flow of entire processingaccording to another exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Some exemplary embodiments are described in detail below with referenceto drawings.

An image inspection system in a first exemplary embodiment includes oneor more imaging devices and an analysis server that processes datatransmitted from the imaging devices. In the following, the analysisserver is also referred to as an image processing apparatus.

FIG. 1A is a diagram illustrating a hardware configuration of theanalysis server (image processing apparatus) according to the presentexemplary embodiment. A central processing unit (CPU) 101 executes acontrol program stored in a read-only memory (ROM) 102, therebyperforming control. A random access memory (RAM) 103 temporarily storesvarious kinds of data provided from each of components. Further, the RAM103 develops a program to be in a state executable by the CPU 101.

A storage unit 104 stores data to be processed according to the presentexemplary embodiment, and stores data acquired from the imaging devicesand an analysis result of the data. As a medium of the storage unit 104,a flash memory, a hard disk drive (HDD), a digital versatile disk(DVD)-RAM, etc. may be used. An input unit 105 includes an operationbutton and a touch panel, and the input unit 105 receives an instructionfrom a worker. An output unit 106 includes an audio speaker and a liquidcrystal panel, and the output unit 106 can output data to be processedand a process result according to the present exemplary embodiment.Further, the image processing apparatus can communicate with the imagingdevices, such as a monitoring camera, through a communication unit 107and can receive captured images.

One analysis server may analyze data of one or a plurality of imagingdevices. Further, the analysis sever and the imaging device may beintegrally configured.

FIG. 1B is a diagram illustrating a software configuration of thesystem. An image management unit 108 has a function of temporallybuffering images 109 captured by the imaging devices, and the imagemanagement unit 108 has a function of recording the captured images 109in the storage unit 104. A work determination unit 110 analyzes theplurality of captured images 109 to determine a start timing and an endtiming of each work, and records the start timing and the end timing asa work determination result 111 in the storage unit 104. An inspectiontarget selection unit 112 selects images and areas on which the imageinspection is to be performed, based on inspection content definitioninformation 113 previously defined and the work determination result111, and records a result of the selection as an inspection targetselection result 114 in the storage unit 104. The inspection targetselection unit 112 includes an inspection target candidate selectionunit that extracts inspection target candidates from the captured images109 and includes a final inspection target selection unit that selects afinal image inspection target from the inspection target candidates.

An image inspection unit 115 performs the image inspection on the imagesselected as the inspection target selection result 114, and the imageinspection unit 115 records a result of the image inspection as an imageinspection result 116 in the storage unit 104. A notification controlunit 117 makes a notification to the worker based on the imageinspection result 116.

The processing corresponding to each operation in the flowchartsdescribed in the exemplary embodiments herein may be achieved bysoftware with use of a CPU, or the processing may be achieved byhardware without software, such as an electronic circuit.

FIG. 2A is a diagram illustrating a using scene of the system accordingto the present exemplary embodiment. In this example, a worker 201performs assembling work on a work object 204 on a workbench 203. Animaging device 202 is installed above the workbench 203, and the imagingdevice 202 captures the motions of the worker 201 and the work object204.

The system confirms whether the work object has been correctly assembledthrough analysis of the image captured by the imaging device 202. In acase where it is determined that there is an assembly defect, the defectis notified to the worker. For example, the defect can be notified tothe worker through output of a voice saying “the screw has beenincorrectly fastened. Please redo the work” using a sentence indicatingthe assembling contents from a speaker.

To assemble a product in a factory, a plurality of assembling works aresequentially performed on one product. The system provides a device thatdetermines whether each work has been executed and measures an executiontime. More specifically, a frame section where each work has beenperformed is determined through analysis of one or more motions of aperson in a frame. Further, the image inspection corresponding to eachassembling work is performed.

FIG. 2B illustrates a relationship between work determination and theimage inspection in the system. In FIG. 2B, frames of the one or moreimages captured by the imaging device 202 are arranged and illustratedin order of capturing time. Frame is included in video image. In thisexample, it is determined that screw fastening work is performed in asection of frames 205. In the present exemplary embodiment, the imageinspection is performed in order to confirm whether the screw has beenfastened at a predetermined position at the completion of the screwfastening work. More specifically, an inspection whether the screw hasbeen fastened at a predetermined position is performed with respect toframes 206 before and after the completion of the screw fastening work.

Frame 207 in FIG. 2C is extracted from frames 206. An inspection targetarea 208 is further selected from within the frame 207. The image of theinspection target area 208 is analyzed to determine whether the assemblyhas been correctly performed.

FIG. 2D illustrates a correct answer image 209 to be referred in theimage inspection. The correct answer image 209 corresponds to theinspection contents and, for example, is an image in a state where thescrew has been correctly fastened. In the image inspection, it can bedetermined whether the assembly has been correctly performed, throughcomparison of the image of the inspection target area 208 and thecorrect answer image 209.

FIG. 2E illustrates a variation of the inspection target area 208. Animage of an inspection target area 210 is an image in the state wherethe screw has been correctly fastened, and an image of an inspectiontarget area 211 is an image in a state where one of two screws has notbeen correctly fastened. When the image of the inspection target area211 is inspected, it is determined that the screw has not been correctlyfastened, and a warning is notified to the worker. As the contents ofthe image inspection, deviation, inclination, etc. at the mountingposition may be inspected in addition to determination of the presenceor absence of a mounting.

FIGS. 3A to 3D are diagrams each illustrating a format of data handledin the system. FIG. 3A illustrates the inspection content definitioninformation 113 that defines associations of the work contents and theinspection contents and that is previously created, for example by amanager of a factory, etc.

A work type 301 is information specifying the work in the assemblingprocess, and an inspection content 302 is information that is associatedwith the work and that specifies inspection contents. For example, aninspection whether a screw has been fastened may be associated with thescrew fastening work. All of the work is not necessarily associated withinspections, and a plurality of inspections may be associated with onework.

An inspection frame definition 303 is definition information specifyinga frame to be inspected corresponding to the inspection contents. Inthis example, four frames before and after the completion of the screwfastening are defined as frames to be inspected for screw fastening.

A reference object 304 and an inspection area definition 305 areinformation specifying the inspection target area within the frame. Aposition of the work object may be moved in the assembling work.Therefore, in some embodiments the inspection target area is defined bya position relative to a specific reference object. The reference object304 is information specifying the reference object, and the inspectionarea definition 305 is information defining a position of the inspectiontarget area by a position relative to the reference object 304. Acorrect answer image 306 is an image in a state where the component tobe inspected has been correctly assembled.

FIG. 3B illustrates the work determination result 111 that is generatedby the system through analysis. A start frame 307 determined as a startof the work, and an end frame 308 determined as an end of the work, arerecorded for each work type 301, which allows for specifying anexecution frame section in units of the work. Each of workers repeatedlyperforms a set of a plurality of work. Therefore, the determinationresult is updated at a time when one set is completed.

FIG. 3C illustrates the inspection target selection result 114 that is aresult of the determination of inspection target candidate frames 309and inspection target candidate areas 310 in the respective frames, fromthe captured image. Further, an inspection image appropriatenessdetermination result 311 is stored as a result of a determinationwhether each of the inspection target candidate areas 310 is appropriatefor an inspection image.

FIG. 3D illustrates the image inspection result 116 in which aninspection result (acceptable or unacceptable) for each inspectioncontent is recorded.

These results of the image analysis may be separately perpetuated in astorage device as necessary. In addition, the captured images 109themselves are stored while a frame number is assigned to each frame.

The above-described operations are described with reference to aflowchart.

FIG. 4 illustrates an outline of an entire processing flow executed bythe image processing apparatus of the system. First, in S401, the oldestimage is acquired from frame images that have been captured by theimaging device and accumulated in the buffer of the system. In S402, awork type is determined. At a timing that each work is first determined,the start frame 307 corresponding to the work is recorded. A method,such as machine learning, can be used for the determination of the worktype. For example, data with a label indicating which work is inprogress is previously prepared for each frame, and a work classifierwhich has learned the data is constructed. The work type can bedetermined for each frame by the classifier. To construct theclassifier, a support vector machine (SVM), a neural network, etc. canbe used.

In S403, it is determined whether one work has been completed. Forexample, in a case where the work determination result is different froma work determination result of a previous frame in comparison, it isdetermined that the work has been completed. The end frame 308 of thecurrent work is recorded in the work determination result 111 at thetiming that it is determined that one work has been completed. If it isdetermined that one work has been completed (YES in S403), theprocessing proceeds to S404, and it is determined whether there is aninspection corresponding to the work by referring to the inspectioncontent definition information 113.

In a case where it is determined in S403 that the work has not beencompleted (NO in S403) or it is determined in S404 that there is noinspection corresponding to the work (NO in S404), the processingreturns to S401, and the work type determination processing is repeated.

In contrast, in a case where it is determined in S404 that there is aninspection corresponding to the work (YES in S404), the processingproceeds to S405. In S405, the inspection target candidate frames areselected. Subsequently, in S406, the inspection target candidate areasare selected from the respective inspection target candidate frames.

In S407, it is determined whether the areas selected in S406 areappropriate to the inspection, and a final inspection target isselected. In S408, the image inspection is performed on the finalinspection target. In S409, it is determined whether the correspondingassembling work has been correctly completed. In a case where it isdetermined that the assembling work has not been correctly completed(YES in S409), notification to the worker is made in S410 to prompt theworker to perform the work again. Details of the processing in S405,S406, and S407 are described below.

This flow is repeatedly performed during execution of the software, andthe flow ends when execution of the software ends.

FIG. 5 is a flowchart illustrating the inspection target candidate frameselection processing in S405. First, in S501, an end frame number of thetarget work is acquired with reference to the work determination result111.

Next, in S502, the frames to be inspected are selected from the capturedimages. In this example, the frame section as the inspection targetcandidate is specified and the frames corresponding to the section areacquired, based on the frame number acquired in 5501 and the definedcontents of the inspection content definition information 113.

The method of specifying the frame section as the inspection targetcandidate according to the present exemplary embodiment is not limitedto the above-described method. The inspection target candidate may be aframe section before the next work is started or a predeterminedpercentage of the section where the target inspection is executed.

FIG. 6 is a flowchart illustrating processing for selecting theinspection target candidate area. In this flow, the areas to beinspected are extracted from the respective inspection target candidateframes selected in the above-described inspection processing forselecting a target candidate frame.

First, in S601, one frame is acquired from the inspection targetcandidate frames. Next, in S602, the frame selected in S601 is analyzedand a reference object defined by the inspection content definitioninformation 113 is recognized. In S602, the position of the work objectcan be determined through template matching with reference to a templatefeature amount for reference object recognition that is previouslydefined. The template matching may be a method in which a template imageis scanned on the input image to calculate a similarity at each positionon the input image, and a position showing the maximum similarity (orwith a similarity equal to or larger than threshold) is detected. Theobject recognition method according to the present exemplary embodimentis not limited to the template matching, and a method such as machinelearning is also usable.

In S603, the inspection area image in the frame is extracted based onthe position of the reference object recognized in S602. In S603, theinspection target area may be extracted based on the position relativeto the reference object previously defined.

In sS604, it is determined whether the processing on all of theinspection target candidate frames has ended. In a case where theprocessing on all of the frames has ended (YES in S604), the processingin this flow ends. Otherwise (NO in S604), the processing is repeatedfrom S601.

FIG. 7 is a flowchart illustrating processing for selecting the finalinspection target in S407. In this flow, it is determined whether theinspection target candidate selected in the flow of FIG. 6 isappropriate as the inspection image, and the image appropriate as theinspection image is selected as the final inspection target.

In S701, one area is acquired from the inspection target areas selectedfrom the plurality of frames. In S702, the presence or absence of ashield at the inspection target position is determined to determinewhether the area selected in S701 is appropriate as the inspectionimage. In the present exemplary embodiment, the assembling site isimaged. Therefore, the inspection target position may be shielded by ahead, an arm, etc. of the worker, and the inspection target position maynot be displayed depending on a frame. Accordingly, the presence orabsence of a shield in the inspection target area in each frame isdetermined, and the inspection target area without a shield is selected.

In a case where it is determined in S702 that there is no shield (NO inS702), the currently-selected area is recorded as the final inspectiontarget in S703. In S704, it is determined whether the determination ofall of the inspection target candidate areas has been made. In a casewhere determination of all of the inspection target candidate areas hasbeen made (YES in S704), the processing in this flow ends. In a casewhere there is an inspection target candidate area which has not beensubjected to the determination (NO in S704), the processing is repeatedfrom S701 for the inspection target candidate area which has not beensubjected to the determination.

FIG. 8 is a diagram illustrating the shield of the inspection targetcandidate area. FIG. 8 illustrates inspection target candidate frames801 to 804, and inspection target candidate areas 805 to 808 arerespectively set in the inspection target candidate frames 801 to 804.Among them, in each of the inspection target candidate areas 805, 806,and 808, the work object in the inspection target candidate area isshielded by the arm of the worker. When such a shield occurs, the stateof the inspection object cannot be correctly recognized, and aninspection whether the assembly has been correctly completed is notaccordingly performable. Therefore, in this example, the inspectiontarget candidate area 807 is most appropriate as the inspection imageand is selected as the final inspection target.

A well-known technology can be used as the shield determination method.For example, a statistic amount of color shade is compared between theinspection target candidate area and the correct answer image, and itmay be determined that shield has occurred in a case where the twostatistic amounts are different by a predetermined amount or more.

Further, as the other method, a range image can be used. A range sensorthat can acquire a distance from an imaging object may be furtherinstalled as an imaging apparatus to acquire the range image for which adistance from the imaging apparatus is measured. A predetermineddistance is previously determined based on an installation position ofthe work object. In a case where there is an object which is nearer tothe imaging apparatus than the predetermined distance, it can bedetermined that the shield occurs.

Alternatively, a contour of an object is extracted from each of theinspection target candidate area and the correct answer image. In a casewhere the contour present in the correct answer image is lacking in theinspection target candidate area, it may be determined that the shieldoccurs.

The processing for selecting the final inspection target according tothe present exemplary embodiment is not limited to the determination ofa presence or absence of the shield. The final inspection target may beselected in consideration of the presence or absence of fog, blur, etc.of the captured area.

After the final inspection target is selected in the flow of FIG. 7, theimage inspection is performed in S408. In the image inspection, thefinal inspection target area and the correct answer image in the statewhere the component has been correctly mounted are compared to determinewhether the component has been correctly mounted in the inspectiontarget. A well-known technology can be used as the method of the imageinspection. For example, a difference from the correct answer image iscalculated, and it is determined as acceptable when the difference isequal to or lower than a predetermined value, and it is determined asunacceptable when the difference is larger than the predetermined value.

In a case where there is a plurality of final inspection targets, thedetermination may be made based on majority of the inspection results,or it may be determined as acceptable when there are one or moreacceptable images.

The contents and the method of the image inspection according to thepresent exemplary embodiment are not limited thereto. The inspectioncontents, correctness of inclination, color shade, etc. of the mountingmay be determined, in addition to the determination of the presence orabsence of the component mounting.

Further, in a case where there is no inspection target candidate areaappropriate for the inspection in the final inspection target selectionprocessing, a notification may be made to the worker to instruct theworker to remove a shielding object. In this case, the inspection can becompleted by performing processing in S405 to S410 on images capturedafter the shielding object is removed.

According to the present exemplary embodiment, it is possible toappropriately determine the inspection timing and the inspection targetarea of the assembled product based on the assembling work in thefactory. This eliminates interrupting the work only for inspection ofthe assembled product. Further, since the time from completion of thework to the inspection is short, it is possible to reduce rework whenthe defect occurs. Further, since the inspection can be executed duringthe assembling work, a state inside the product that cannot be inspectedafter completion of the whole assembly can be inspected.

A work unit corresponding to the work according to the present exemplaryembodiment can be freely determined by a manager of the factory. Theunit may be an operation to mount one component, or operations to mounta plurality of components may be collected as one work.

A second exemplary embodiment is described below. A description of theconfigurations that are the same as the configurations according to thefirst exemplary embodiment is omitted.

In the first exemplary embodiment, the imaging is performed at theconstant frame rate. However, the work in the assembling process may beperformed at an extremely high speed, and the image without shield maynot be captured at a time that the inspection is to be performed, if theimaging is carried out at the constant frame rate. In contrast, if theimaging is performed constantly at a high frame rate, a load on thesystem is increased. In the present exemplary embodiment, a unitconfigured to change the frame rate of the imaging device is provided,and the frame rate is increased only when inspection timing comes near.

FIG. 9 illustrates a flow outline of the processing executed by theimage processing apparatus of the system according to the secondexemplary embodiment. The processing in S901 to S903 and in S908 to S913are the same as or similar to the processing in S401 to S403 and in S405to S410 in FIG. 4 according to the first exemplary embodiment. First, inS901, the oldest image is acquired from frame images that have beencaptured by the imaging device and accumulated in the buffer of thesystem. In S902, the work type is determined. In S903, it is determinedwhether one work has been completed.

When it is determined that one work has been completed (YES in S903),the processing proceeds to S904, and it is determined whether there isan inspection corresponding to “work next to the completed work”. If itis determined in S904 that there is an inspection corresponding to thenext work (YES in S904), the processing proceeds to S905, and aninstruction to increase the frame rate is issued to the imaging device.In contrast, if it is determined in S904 that there is no inspectioncorresponding to the next work (NO in S904), the processing proceeds toS906, and the frame rate is set to a normal value.

Next, the processing proceeds to S907, and it is determined whetherthere is an inspection corresponding to the work, the completion ofwhich has been determined in S903. In a case where it is determined inS907 that there is a corresponding inspection (YES in S907), theprocessing proceeds to S908, and the processing for selecting theinspection target candidate frames is performed. The processing in andafter S908 are the same as or similar to the processing in the firstexemplary embodiment.

This flow is repeated to perform the imaging control in which the framerate is made higher than the frame rate in the normal state in the casewhere there is an inspection corresponding to the next work, andotherwise, the frame rate is returned to the frame rate in the normalstate. This makes it possible to increase the frame rate only near theinspection timing.

Although the image inspection is executed immediately after theexecution of the work ends in the present exemplary embodiment, theconfiguration is not limited thereto, and the inspection timing may bedelayed in consideration of a calculation load on the system.

For example, when a cover is mounted after five works from the screwfastening work, even if the determination of the screw fastening isperformed before the cover mounting work, the screw fastening work isperformable again without rework to detach the cover. As describedabove, the inspection execution timing may be previously designated bythe manager of the factory, and the inspection may be controlled to beperformed before the designated timing. For example, since theprocessing load is high while the frame rate of the imaging is increasedin the present exemplary embodiment, performing the image inspection inthe other section makes it possible to smooth the load on the system.

Other Embodiments

Some other exemplary embodiments are described below. Description of theconfigurations same as the configurations according to the firstexemplary embodiment is omitted.

The work execution determination method according to the first andsecond exemplary embodiments sequentially determines the work type foreach frame. The work execution determination method according to someexemplary embodiments, however, is not limited to this. For example, achange point after the plurality of works have been completed may bedetected in one process, and the work execution determination may beperformed collectively on the plurality of works.

In these exemplary embodiments, a group of the plurality of works forthe work execution determination is defined as a “phase”. To enable thesystem to recognize the change point of the phase, the phase is changedat a time that the position of the object is changed. For example, in acase where it is known that there is movement to reverse and rotate thework object 204 between works 4 and 5 and between works 6 and 7 in theprocess of works 1 to 9, three phases of works 1 to 4, works 5 and 6,and works 7 to 9 can be defined.

The execution determination of the work in the phase and the imageinspection are performed at the change point of the phase, andnotification is made in a case where a defect is detected. In theabove-described example, determination and an inspection for the works 1to 4 are performed when the phase is changed from the first phase to thesecond phase.

FIG. 10 illustrates an example embodiment of a flow outline of entireprocessing executed by the image processing apparatus of the system whenthe work execution determination is made for each phase. Processing inS1007 to S1012 is the same as or similar to processing in S405 to S410in FIG. 4 according to the first exemplary embodiment.

First, in S1001, the oldest image is acquired from the frame images thathave been captured by the imaging device and accumulated in the bufferof the system. In S1002, it is determined whether one phase has beencompleted. In S1002, the presence or absence of a positional change ofthe work object can be determined through template matching withreference to a template feature amount in a phase change that ispreviously defined. A phase start determination method according to thepresent exemplary embodiment is not limited to this method, and a methodsuch as machine learning is also usable. Processing in S1001 and S1002is repeatedly performed until it is determined that the phase has beencompleted.

As a specific algorism for the machine learning, for example a mostneighborhood method, a naïve B ayes method, a decision tree method, anda support vector machine, may be used. Further, a deep learning thatuses a neural network to generate a feature amount for learning and acoupling weighting factor by itself may be also used.

In S1003, execution of the work in the phase is determined. For example,a frame and the work which has been performed can be determined byanalyzing the temporal change of the feature amount for each framewithin the phase and the change of a passing position of a hand of theworker displayed in the frame. In S1003, the start frame 307 and the endframe 308 of the work are determined for the plurality of works withinthe phase and are recorded.

In S1004 to S1013, the processing is repeated for each work in thephase. First, in S1005, one work is selected from the work in the phase.In S1006, it is determined whether there is an inspection correspondingto the selected work. In a case where there is no correspondinginspection (NO in S1006), the processing returns to S1005, and theprocessing is repeated.

In S1007 to S1012, the inspection target candidate frames, theinspection target candidate areas, and the final inspection target areselected, the image inspection is executed, and a defect is notified, ina manner that is the same as or similar to S405 to S410 in FIG. 4. Theabove-described processing is repeated for the number of works withinthe phase. In a case where it is determined in S1013 that the inspectionof all of the works in the phase has been completed, the processingreturns to S1001.

As described above, the work determination is collectively performed onthe frames within the phase, and the work execution determination can bemade in consideration of a relationship before and after the work and ofthe temporal change of the motion of the worker.

The processing by the work determination unit, the image inspectionunit, etc. among the above-described processing units may be performedby a learnt model that has performed machine learning, in place of theprocessing units. In this case, for example, a plurality of combinationsof input data and output data for the processing units is prepared aslearning data, and the learnt model is created and acquires knowledgefrom the leaning data through machine learning and outputs, as a result,output data corresponding to input data based on the acquired knowledge.The learnt model can be configured by, for example, a neural networkmodel. Further, the learnt model operates as a program to performprocessing equivalent to the processing by the above-describedprocessing units in cooperation with a CPU, a graphics processing unit(GPU), etc., thereby performing the processing of the processing units.The above-described learnt model may be updated after predeterminedprocessing, as necessary.

Some embodiments are achieved by supplying software (program) achievingthe functions of the above-described exemplary embodiments to a systemor an apparatus through a network or various kinds of storage media, andcausing a computer (or CPU, micro processing unit (MPU), etc.) of thesystem or the apparatus to read out and execute the program.

According to these exemplary embodiments, it is possible to inspectmounting of a component in an assembled product at appropriate timing.

Other Embodiments

Some embodiment(s) can also be realized by a computer of a system orapparatus that reads out and executes computer-executable instructions(e.g., one or more programs) recorded on a storage medium (which mayalso be referred to more fully as a ‘non-transitory computer-readablestorage medium’) to perform the functions of one or more of theabove-described embodiment(s) and/or that includes one or more circuits(e.g., application specific integrated circuit (ASIC)) for performingthe functions of one or more of the above-described embodiment(s), andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer-executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s) and/or controlling the one or morecircuits to perform the functions of one or more of the above-describedembodiment(s). The computer may comprise one or more processors (e.g.,central processing unit (CPU), micro processing unit (MPU)) and mayinclude a network of separate computers or separate processors to readout and execute the computer executable instructions. Thecomputer-executable instructions may be provided to the computer, forexample, from a network or the storage medium. The storage medium mayinclude, for example, one or more of a hard disk, a random-access memory(RAM), a read only memory (ROM), a storage of distributed computingsystems, an optical disk (such as a compact disc (CD), digital versatiledisc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memorycard, and the like.

While the present disclosure has described exemplary embodiments, it isto be understood that some embodiments are not limited to the disclosedexemplary embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims priority to Japanese Patent Application No.2018-087492, which was filed on Apr. 27, 2018 and which is herebyincorporated by reference herein in its entirety.

What is claimed is:
 1. An image processing apparatus that performs aninspection on an assembly component with use of one or more images ofassembling work, the image processing apparatus comprising: a workdetermination unit configured to determine, from the one or more images,one or both of a start frame and an end frame of an assembling workcaptured at a change point of the assembling work; a selection unitconfigured to select frames to be inspected from the one or more imagesbased on a result of the determination by the work determination unit;and an inspection unit configured to perform an inspection on the framesselected by the selection unit.
 2. The image processing apparatusaccording to claim 1, wherein the selection unit selects a candidatearea to be inspected from each of the selected frames.
 3. The imageprocessing apparatus according to claim 1, wherein the workdetermination unit determines one or both of the start frame and the endframe for each work type, and wherein the selection unit selects aninspection target for each work type determined by the workdetermination unit.
 4. The image processing apparatus according to claim2, wherein an appropriateness as an inspection image is determined foreach candidate area to be inspected that is selected from each of theframes to be inspected by the selection unit, and an area with a highappropriateness is selected as the inspection target.
 5. The imageprocessing apparatus according to claim 4, wherein the selection unitdetermines a presence or absence of a shield in each of the candidateareas to be inspected, and selects an area image showing less of theshield as the inspection target.
 6. The image processing apparatusaccording to claim 1, wherein the selection unit further includes anotification unit configured to make a notification in a case wherethere is no image selected as the inspection target.
 7. The imageprocessing apparatus according to claim 1, further comprising an imagingcontrol unit configured to control an imaging apparatus to change aframe rate based on a result of the determination by the workdetermination unit.
 8. An image processing method to perform aninspection on an assembly component with use of one or more images of anassembling work, the method comprising: determining, from the one ormore images, one or both of a start frame and an end frame of anassembling work captured at a change point of the assembling work;selecting frames to be inspected from the one or more images based on aresult of the determination; and performing an inspection on theselected frames.
 9. The image processing method according to claim 8,wherein an area to be inspected is selected from each of the selectedframes, to generate a candidate area to be inspected.
 10. The imageprocessing method according to claim 8, wherein one or both of the startframe and the end frame are determined for each work type, and aninspection target is selected for each determined work type.
 11. Theimage processing method according to claim 9, wherein an appropriatenessas an inspection image is determined for each candidate area to beinspected that is selected from each of the frames to be inspected, andan area with a high appropriateness is selected as the inspectiontarget.
 12. The image processing method according to claim 11, wherein apresence or absence of a shield in each of the candidate areas to beinspected is determined, and an area image showing less of the shield isselected as the inspection target.
 13. The image processing methodaccording to claim 8, wherein the selecting further includes making anotification in a case where there is no image selected as theinspection target.
 14. The image processing method according to claim 8,further comprising controlling an imaging apparatus to change a framerate based on a result of the work determination.
 15. The imageprocessing method according to claim 8, wherein the work determinationis made while the assembling work of the component is in progress, andwherein the inspection is performed while the assembling work of thecomponent is in progress.
 16. The image processing method according toclaim 8, wherein the work determination is made before completion of theassembling work of the component, and wherein the inspection isperformed before completion of the assembling work of the component. 17.The image processing method according to claim 8, wherein the workdetermination is made through machine learning.
 18. The image processingmethod according to claim 8, wherein the inspection is performed throughmachine learning.
 19. The image processing method according to claim 16,wherein a work type of the work is screw fastening.
 20. A non-transitorycomputer-readable storage medium storing a computer-executable programfor causing a computer to perform an image processing method to performan inspection on an assembly component with use of one or more images ofan assembling work, the method comprising: determining, from the one ormore images, one or both of a start frame and an end frame of anassembling work captured at a change point of the assembling work;selecting frames to be inspected from the one or more images based on aresult of the determination; and performing an inspection on theselected frames.